Multivariate Time Series Data Research Articles

Anomaly Detection for Asynchronous Multivariate Time Series of Nuclear Power Plants Using a Temporal-Spatial Transformer

Industrial process monitoring is a critical application of multivariate time-series (MTS) anomaly detection, especially crucial for safety-critical systems such as nuclear power plants (NPPs). However, some current data-driven process monitoring approaches may not fully capitalize on the temporal-spatial correlations inherent in operational MTS data. Particularly, asynchronous time-lagged correlations may exist among variables in actual NPPs, which further complicates this challenge. In this work, a reconstruction-based MTS anomaly detection approach based on a temporal-spatial transformer is proposed. It employs a two-stage temporal-spatial attention mechanism combined with a multi-scale strategy to learn the dependencies within normal operational data at various scales, thereby facilitating the extraction of temporal-spatial correlations from asynchronous MTS. Experiments on simulated datasets and real NPP datasets demonstrate that the proposed model possesses stronger feature learning capabilities, as evidenced by its improved performance in signal reconstruction and anomaly detection for asynchronous MTS data. Moreover, the proposed TS-Trans model enables earlier detection of anomalous events, which holds significant importance for enhancing operational safety and reducing potential losses in NPPs.

Using transformer and a reweighting technique to develop a remaining useful life estimation method for turbofan engines

Estimating the Remaining Useful Life (RUL) of industrial machinery is an important task in Prognostics and Health Management (PHM). Accurate RUL prediction based on current health status and real-time sensor measurements can help establish efficient maintenance strategies in advance, thus improving the productivity and efficiency of machinery operations. However, considering the nature of industrial machinery, a data imbalance problem, which exists in current RUL estimation datasets like the Commercial Modular Aero-Propulsion System Simulation (C-MAPSS) dataset, hinders data-driven approaches. To address the detrimental data imbalance problem and improve estimation performance, this research proposes an Adaptive RUL-wise Reweighting (ARR) technique. ARR uses kernel smoothing to maintain the inherent continuity of RUL and a reweighting method to rebalance the effects of samples with different ground-truth RULs. In addition, this research proposes a novel RUL estimation method that uses a transformer architecture, which has not yet been widely applied for PHM tasks. Using a transformer-based estimation model with high expressive power capable of handling multivariate time-series data has proven effective in RUL estimation. Comprehensive experiments using the C-MAPSS dataset based on cross validation are conducted to find optimal hyperparameter configurations and to validate the proposed method’s effectiveness in RUL estimation. For subsets FD001 and FD003 of the C-MAPSS dataset, the proposed method provides state-of-the-art estimation performance with the lowest Root Mean Squared Error (RMSE) values of 11.36 and 11.28 and score values of 192.22 and 133.41, respectively. The proposed method also shows highly competitive performance on subsets FD002 and FD004 of the C-MAPSS dataset.

DTAAD: Dual Tcn-attention networks for anomaly detection in multivariate time series data

Anomaly detection techniques enable effective anomaly detection and diagnosis in multi-variate time series data, which are of major significance for today’s industrial applications. However, establishing an anomaly detection system that can be rapidly and accurately located is a challenging problem due to the lack of anomaly labels, the high dimensional complexity of the data, memory bottlenecks in actual hardware, and the need for fast reasoning. In this paper, we propose an anomaly detection and diagnosis model, DTAAD, based on Transformer and Dual Temporal Convolutional Network (TCN). Our overall model is an integrated design in which an autoregressive model (AR) combines with an autoencoder (AE) structure. Scaling methods and feedback mechanisms are introduced to improve prediction accuracy and expand correlation differences. Constructed by us, the Dual TCN-Attention Network (DTA) uses only a single layer of Transformer encoder in our baseline experiment, belonging to an ultra-lightweight model. Our extensive experiments on seven public datasets validate that DTAAD exceeds the majority of currently advanced baseline methods in both detection and diagnostic performance. Specifically, DTAAD improved F1 scores by 8.38% and reduced training time by 99% compared to the baseline. The code and training scripts are publicly available on GitHub at https://github.com/Yu-Lingrui/DTAAD.

A Deep Learning Approach to Predict Bleeding Risk Over Time in Patients on Extended Anticoagulation Therapy

BackgroundThus far, all the clinical models developed to predict major bleeding in patients on extended anticoagulation therapy use the baseline predictors to stratify patients into different risk groups. Therefore, these models do not account for the clinical changes and events that occur after the baseline visit, which can modify risk of bleeding. However, it is difficult to develop predictive models from the routine follow-up clinical interviews which are irregular sequences of multivariate time series data. ObjectivesTo demonstrate that deep learning can incorporate patient time-series follow-up data to improve prediction of major bleeding. MethodWe used the baseline and follow-up data that was collected over 8 years in a longitudinal cohort study of 2542 patients, of whom 118 had major bleeding. Four supervised neural network-based machine learning models were trained on the baseline, or the follow-up, or both datasets on 70% of the data. The performance of these models was evaluated, along with modified versions of previously developed clinical models (CHAP, ACCP, RIETE, VTE-BLEED, HAS-BLED, and OBRI), on the remaining 30% of the data. ResultsAn ensemble of feedforward and recurrent neural networks that used the baseline and follow-up data was the best-performing model, achieving a sensitivity and a specificity of 61% and 82%, respectively, in identifying major bleeding, and it outperformed the previously developed clinical models in terms of area under the ROC curve (82%) and area under the precision-recall curve (14%). ConclusionTime series follow-up data can improve major bleeding prediction in patients on extended anticoagulation therapy.

DPDGAD: A Dual-Process Dynamic Graph-based Anomaly Detection for multivariate time series analysis in cyber-physical systems

With the rapid development of cyber–physical systems, an increasing amount of data is stored in the form of multivariate time series. Detecting anomalies within multivariate time series has become a crucial means to ensure the proper functioning of cyber–physical systems. Traditional methods for anomaly detection in multivariate time series often categorize features into temporal and spatial features based on their representative dimensions. However, these approaches tend to overlook the unique characteristics of cyber–physical systems reflected in the multivariate time series. This hinders their ability to perform well in cases with complex multivariate time series data. To overcome this challenge, we introduce a novel approach that classifies the spatiotemporal features of multivariate time series data into steady features (features that remain constant under external variations) and dynamic features (features that respond to external changes), aligning with the nature of cyber–physical systems. Addressing this challenge involves two main aspects: (1) determining how to classify steady and dynamic features in multivariate time series data and (2) leveraging these features for anomaly detection.To address these challenges, we propose a Dual-Process Dynamic Graph-based time series Anomaly Detection (DPDGAD) method. DPDGAD enhances the generalizability of anomaly detection in complex spatiotemporal environments and improves the interpretability of the detected anomalies. We validate the performance of DPDGAD through six real-world datasets generated by cyber–physical systems, achieving F1 scores of 0.7529 and 0.9009 on WADI and SWaT, respectively.

HFN: Heterogeneous feature network for multivariate time series anomaly detection

As the key step of anomaly detection for multivariate time-series (MTS) data, learning the relations among different variables has been explored by many approaches. However, most existing approaches overlook the heterogeneity among variables, that is, different types of variables (continuous numerical variables, discrete categorical variables or hybrid variables) may have different edge distributions. In this paper, we propose a novel semi-supervised anomaly detection framework based on a heterogeneous feature network (HFN) for MTS. Specifically, we first combine the embedding similarity subgraph generated by sensor embedding and the feature value similarity subgraph generated by sensor values to construct a time-series heterogeneous graph, which fully utilizes the rich heterogeneous mutual information among variables. Then, a prediction model containing nodes and channel attentions is jointly optimized to obtain better time-series representations. This approach fuses the state-of-the-art technologies of heterogeneous graph structure learning (HGSL) and representation learning. Experimental results on four sensor datasets from real-world applications demonstrate that our approach achieves more accurate anomaly detection compared to baseline methods, laying a foundation for the rapid positioning of anomalies.

Detection of quadratic phase coupling by cross-bicoherence and spectral Granger causality in bifrequencies interactions

Quadratic Phase Coupling (QPC) serves as an essential statistical instrument for evaluating nonlinear synchronization within multivariate time series data, especially in signal processing and neuroscience fields. This study explores the precision of QPC detection using numerical estimates derived from cross-bicoherence and bivariate Granger causality within a straightforward, yet noisy, instantaneous multiplier model. It further assesses the impact of accidental statistically significant bifrequency interactions, introducing new metrics such as the ratio of bispectral quadratic phase coupling and the ratio of bivariate Granger causality quadratic phase coupling. Ratios nearing 1 signify a high degree of accuracy in detecting QPC. The coupling strength between interacting channels is identified as a key element that introduces nonlinearities, influencing the signal-to-noise ratio in the output channel. The model is tested across 59 experimental conditions of simulated recordings, with each condition evaluated against six coupling strength values, covering a wide range of carrier frequencies to examine a broad spectrum of scenarios. The findings demonstrate that the bispectral method outperforms bivariate Granger causality, particularly in identifying specific QPC under conditions of very weak couplings and in the presence of noise. The detection of specific QPC is crucial for neuroscience applications aimed at better understanding the temporal and spatial coordination between different brain regions.

MTS2Graph: Interpretable multivariate time series classification with temporal evolving graphs

Conventional time series classification approaches based on bags of patterns or shapelets face significant challenges in dealing with a vast amount of feature candidates from high-dimensional multivariate data. In contrast, deep neural networks can learn low-dimensional features efficiently, and in particular, convolutional neural networks have shown promising results in classifying multivariate time series data. A key factor in the success of deep neural networks is this astonishing expressive power. However, this power comes at the cost of complex, black-boxed models, conflicting with the goals of building reliable and human-understandable models. In this work11The code can be found at https://github.com/raneeny/MTS2Graph., we introduce a new interpretable framework for multivariate time series data that by extracting and clustering the input quantifies the contribution of time-varying input variables and each signal’s role to the classification. We construct a graph that captures the temporal relationship between the extracted patterns for each layer and propose an effective merging strategy to aggregate those graphs into one. Finally, a graph embedding algorithm generates new representations of the created interpretable time-series features. Our extensive experiments indicate the benefit of our time-aware graph-based representation in multivariate time series classification while enriching them with more interpretability.

MGAD: Mutual Information and Graph Embedding Based Anomaly Detection in Multivariate Time Series

MGAD: Mutual Information and Graph Embedding Based Anomaly Detection in Multivariate Time Series

Long-Term Forecasting Using MAMTF: A Matrix Attention Model Based on the Time and Frequency Domains

There are many time series forecasting methods, but there are few research methods for long-term multivariate time series forecasting, which are mainly dominated by a series of forecasting models developed on the basis of a transformer. The aim of this study is to perform forecasting for multivariate time series data and to improve the forecasting accuracy of the model. In the recent past, it has appeared that the prediction effect of linear models surpasses that of the family of self-attention mechanism models, which encourages us to look for new methods to solve the problem of long-term multivariate time series forecasting. In order to overcome the problem that the temporal order of information is easily broken in the self-attention family and that it is difficult to capture information on long-distance data using recurrent neural network models, we propose a matrix attention mechanism, which is able to weight each previous data point equally without breaking the temporal order of the data, so that the overall data information can be fully utilized. We used the matrix attention mechanism as the basic module to construct the frequency domain block and time domain block. Since complex and variable seasonal component features are difficult to capture in the time domain, mapping them to the frequency domain reduces the complexity of the seasonal components themselves and facilitates data feature extraction. Therefore, we use the frequency domain block to extract the seasonal information with high randomness and poor regularity to help the model capture the local dynamics. The time domain block is used to extract the smooth floating trend component information to help the model capture long-term change patterns. This also improves the overall prediction performance of the model. It is experimentally demonstrated that our model achieves the best prediction results on three public datasets and one private dataset.

Enhancing Air Quality Forecasting: A Novel Spatio-Temporal Model Integrating Graph Convolution and Multi-Head Attention Mechanism

Forecasting air quality plays a crucial role in preventing and controlling air pollution. It is particularly significant for improving preparedness for heavily polluted weather conditions and ensuring the health and safety of the population. In this study, a novel deep learning model for predicting air quality spatio-temporal variations is introduced. The model, named graph long short-term memory with multi-head attention (GLSTMMA), is designed to capture the temporal patterns and spatial relationships within multivariate time series data related to air quality. The GLSTMMA model utilizes a hybrid neural network architecture to effectively learn the complex dependencies and correlations present in the data. The extraction of spatial features related to air quality involves the utilization of a graph convolutional network (GCN) to collect air quality data based on the geographical distribution of monitoring sites. The resulting graph structure is imported into a long short-term memory (LSTM) network to establish a Graph LSTM unit, facilitating the extraction of temporal dependencies in air quality. Leveraging a Graph LSTM unit, an encoder-multiple-attention decoder framework is formulated to enable a more profound and efficient exploration of spatio-temporal correlation features within air quality time series data. The research utilizes the 2019–2021 multi-source air quality dataset of Qinghai Province for experimental assessment. The results indicate that the model effectively leverages the impact of multi-source data, resulting in optimal accuracy in predicting six air pollutants.

Fully-Connected Spatial-Temporal Graph for Multivariate Time-Series Data

Multivariate Time-Series (MTS) data is crucial in various application fields. With its sequential and multi-source (multiple sensors) properties, MTS data inherently exhibits Spatial-Temporal (ST) dependencies, involving temporal correlations between timestamps and spatial correlations between sensors in each timestamp. To effectively leverage this information, Graph Neural Network-based methods (GNNs) have been widely adopted. However, existing approaches separately capture spatial dependency and temporal dependency and fail to capture the correlations between Different sEnsors at Different Timestamps (DEDT). Overlooking such correlations hinders the comprehensive modelling of ST dependencies within MTS data, thus restricting existing GNNs from learning effective representations. To address this limitation, we propose a novel method called Fully-Connected Spatial-Temporal Graph Neural Network (FC-STGNN), including two key components namely FC graph construction and FC graph convolution. For graph construction, we design a decay graph to connect sensors across all timestamps based on their temporal distances, enabling us to fully model the ST dependencies by considering the correlations between DEDT. Further, we devise FC graph convolution with a moving-pooling GNN layer to effectively capture the ST dependencies for learning effective representations. Extensive experiments show the effectiveness of FC-STGNN on multiple MTS datasets compared to SOTA methods. The code is available at https://github.com/Frank-Wang-oss/FCSTGNN.

Graph-Aware Contrasting for Multivariate Time-Series Classification

Contrastive learning, as a self-supervised learning paradigm, becomes popular for Multivariate Time-Series (MTS) classification. It ensures the consistency across different views of unlabeled samples and then learns effective representations for these samples. Existing contrastive learning methods mainly focus on achieving temporal consistency with temporal augmentation and contrasting techniques, aiming to preserve temporal patterns against perturbations for MTS data. However, they overlook spatial consistency that requires the stability of individual sensors and their correlations. As MTS data typically originate from multiple sensors, ensuring spatial consistency becomes essential for the overall performance of contrastive learning on MTS data. Thus, we propose Graph-Aware Contrasting for spatial consistency across MTS data. Specifically, we propose graph augmentations including node and edge augmentations to preserve the stability of sensors and their correlations, followed by graph contrasting with both node- and graph-level contrasting to extract robust sensor- and global-level features. We further introduce multi-window temporal contrasting to ensure temporal consistency in the data for each sensor. Extensive experiments demonstrate that our proposed method achieves state-of-the-art performance on various MTS classification tasks. The code is available at https://github.com/Frank-Wang-oss/TS-GAC.

AutoMixer for Improved Multivariate Time-Series Forecasting on Business and IT Observability Data

The efficiency of business processes relies on business key performance indicators (Biz-KPIs), that can be negatively impacted by IT failures. Business and IT Observability (BizITObs) data fuses both Biz-KPIs and IT event channels together as multivariate time series data. Forecasting Biz-KPIs in advance can enhance efficiency and revenue through proactive corrective measures. However, BizITObs data generally exhibit both useful and noisy inter-channel interactions between Biz-KPIs and IT events that need to be effectively decoupled. This leads to suboptimal forecasting performance when existing multivariate forecasting models are employed. To address this, we introduce AutoMixer, a time-series Foundation Model (FM) approach, grounded on the novel technique of channel-compressed pretrain and finetune workflows. AutoMixer leverages an AutoEncoder for channel-compressed pretraining and integrates it with the advanced TSMixer model for multivariate time series forecasting. This fusion greatly enhances the potency of TSMixer for accurate forecasts and also generalizes well across several downstream tasks. Through detailed experiments and dashboard analytics, we show AutoMixer's capability to consistently improve the Biz-KPI's forecasting accuracy (by 11-15%) which directly translates to actionable business insights.

Multivariate Time-Series Imagification with Time Embedding in Constrained Environments (Student Abstract)

We present an imagification approach for multivariate time-series data tailored to constrained NN-based forecasting model training environments. Our imagification process consists of two key steps: Re-stacking and time embedding. In the Re-stacking stage, time-series data are arranged based on high correlation, forming the first image channel using a sliding window technique. The time embedding stage adds two additional image channels by incorporating real-time information. We evaluate our method by comparing it with three benchmark imagification techniques using a simple CNN-based model. Additionally, we conduct a comparison with LSTM, a conventional time-series forecasting model. Experimental results demonstrate that our proposed approach achieves three times faster model training termination while maintaining forecasting accuracy.

A multi-granularity hierarchical network for long- and short-term forecasting on multivariate time series data

Multivariate time series forecasting is a significant research problem in many fields such as economics, finance and transportation, where simultaneous long- and short-term forecasting is required. However, current techniques are typically limited to a single short-term or a long-term forecast. To address the limitation, a novel multi-granularity hierarchical network, GNet-LS, is proposed for long- and short-term forecasting on multivariate time series data, which takes into account the separate role of internal correlation and external relationship. First, the original time series sequence is divided into multiple granular sequences based on downsampling, to reduce error accumulation caused by long-term prediction. In order to discover the external relationships between variables, the CNN module slides over the sequence of variables. The global CNN and local CNN are built to implement periodic and nonperiodic extraction, respectively. Next, a self-attention module is used to model dependencies between the output of local CNN and global CNN. The LSTM networks and attention mechanisms are used to mine internal correlation of the target variable on time series. Then, multiple granular external relationships and internal correlation are obtained in parallel. Finally, external relationships and internal correlation are fused together by splicing and overlay to obtain both long-term and short-term forecasts. The experimental results demonstrate that the proposed GNet-LS outperforms a bunch of compared methods in terms of RSE, CORR, MAE and RMSE.

Identifying Flare-indicative Photospheric Magnetic Field Parameters from Multivariate Time-series Data of Solar Active Regions

Photospheric magnetic field parameters are frequently used to analyze and predict solar events. Observation of these parameters over time, i.e., representing solar events by multivariate time-series (MVTS) data, can determine relationships between magnetic field states in active regions and extreme solar events, e.g., solar flares. We can improve our understanding of these events by selecting the most relevant parameters that give the highest predictive performance. In this study, we propose a two-step incremental feature selection method for MVTS data using a deep-learning model based on long short-term memory (LSTM) networks. First, each MVTS feature (magnetic field parameter) is evaluated individually by a univariate sequence classifier utilizing an LSTM network. Then, the top performing features are combined to produce input for an LSTM-based multivariate sequence classifier. Finally, we tested the discrimination ability of the selected features by training downstream classifiers, e.g., Minimally Random Convolutional Kernel Transform and support vector machine. We performed our experiments using a benchmark data set for flare prediction known as Space Weather Analytics for Solar Flares. We compared our proposed method with three other baseline feature selection methods and demonstrated that our method selects more discriminatory features compared to other methods. Due to the imbalanced nature of the data, primarily caused by the rarity of minority flare classes (e.g., the X and M classes), we used the true skill statistic as the evaluation metric. Finally, we reported the set of photospheric magnetic field parameters that give the highest discrimination performance in predicting flare classes.

Developing a multivariate time series forecasting framework based on stacked autoencoders and multi-phase feature

Time series forecasting across different domains has received massive attention as it eases intelligent decision-making activities. Recurrent neural networks and various deep learning algorithms have been applied to modeling and forecasting multivariate time series data. Due to intricate non-linear patterns and significant variations in the randomness of characteristics across various categories of real-world time series data, achieving effectiveness and robustness simultaneously poses a considerable challenge for specific deep-learning models. We have proposed a novel prediction framework with a multi-phase feature selection technique, a long short-term memory-based autoencoder, and a temporal convolution-based autoencoder to fill this gap. The multi-phase feature selection is applied to retrieve the optimal feature selection and optimal lag window length for different features. Moreover, the customized stacked autoencoder strategy is employed in the model. The first autoencoder is used to resolve the random weight initialization problem. Additionally, the second autoencoder models the temporal relation between non-linear correlated features with convolution networks and recurrent neural networks.Finally, the model's ability to generalize, predict accurately, and perform effectively is validated through experimentation with three distinct real-world time series datasets. In this study, we conducted experiments on three real-world datasets: Energy Appliances, Beijing PM2.5 Concentration, and Solar Radiation. The Energy Appliances dataset consists of 29 attributes with a training size of 15,464 instances and a testing size of 4239 instances. For the Beijing PM2.5 Concentration dataset, there are 18 attributes, with 34,952 instances in the training set and 8760 instances in the testing set. The Solar Radiation dataset comprises 11 attributes, with 22,857 instances in the training set and 9797 instances in the testing set. The experimental setup involved evaluating the performance of forecasting models using two distinct error measures: root mean square error and mean absolute error. To ensure robust evaluation, the errors were calculated at the identical scale of the data. The results of the experiments demonstrate the superiority of the proposed model compared to existing models, as evidenced by significant advantages in various metrics such as mean squared error and mean absolute error. For PM2.5 air quality data, the proposed model's mean absolute error is 7.51 over 12.45, about ∼40% improvement. Similarly, the mean square error for the dataset is improved from 23.75 to 11.62, which is ∼51%of improvement. For the solar radiation dataset, the proposed model resulted in ∼34.7% improvement in means squared error and ∼75% in mean absolute error. The recommended framework demonstrates outstanding capabilities in generalization and outperforms datasets spanning multiple indigenous domains.

Clinical deep model to analyse medical multivariate time-series data for health diagnosis

ABSTRACT Clinical auxiliary decision-making is related to life and health of patients, so the deep model needs to extract the personalised representation of patients to ensure high analysis and prediction accuracy; and provide a basis for prediction conclusions. In this context, a clinical deep model proposed an interpretable assessment method of patient health status based on contextual learning of medical features, encoding the time-series features of each variable separately, and using a multi-head de-coordination self-attention mechanism for learning Relationships between different features; feature skip-connection encoding based on a compressed excitation mechanism is proposed to improve the sensitivity of the model.

DSDCLNet: Dual-stream encoder and dual-level contrastive learning network for supervised multivariate time series classification

In recent years, deep learning approaches have shown remarkable advancements in Multivariate Time Series Classification (MTSC) tasks. However, the existing approaches primarily focus on capturing the long-term correlations of time series or identifying local key sequence fragments, inevitably neglecting the synergistic properties between global and local components. Additionally, most representation learning methods for MTSC rely on self-supervised learning, which limits their ability to fully exploit label information. Hence, this paper proposes a novel approach termed Dual-Stream Encoder and Dual-Level Contrastive Learning Network (DSDCLNet), which integrates a Dual-Stream Encoder (DSE) and Dual-level Contrastive Learning (DCL). First, to extract multiscale local-global features from multivariate time series data, we employ a DSE architecture comprising an Attention-Gated Recurrent Unit (AGRU) and a Dual-layer Multiscale Convolutional Neural Network (DMSCNN). Specifically, DMSCNN consists of a series of multi-scale convolutional layers and a max pooling layer. Second, to maximize the utilization of label information, a new loss function is designed, which combines classification loss, instance-level contrastive loss, and temporal-level contrastive loss. Finally, experiments are conducted on the UEA datasets and the results demonstrate that DSDCLNet achieves the highest average accuracy of 0.77, outperforming traditional approaches, deep learning approaches, and self-supervised approaches on 30, 23, and 27 datasets, respectively.